2024-11-05 18:13:23,937 [ 231642 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2024-11-05 18:13:23,938 [ 231642 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2024-11-05 18:13:23,938 [ 231642 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2024-11-05 18:13:23,938 [ 231642 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_7dcml6 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1d1c87eee2db -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=1 --color=no --durations=0 test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: repeat-0.9.3, reportlog-0.4.0, random-0.2, timeout-2.2.0, order-1.0.1, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 2 items test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users FAILED [ 50%] test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total SKIPPED [100%] =================================== FAILURES =================================== _______ test_secure_cluster_distributed_over_distributed_different_users _______ def test_secure_cluster_distributed_over_distributed_different_users(): # This works because we will have initial_user='default' > n1.query( "SELECT * FROM remote('n1', currentDatabase(), dist_secure)", user="new_user" ) test_distributed_inter_server_secret/test.py:416: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3451: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Received exception from server (version 24.3.12): E Code: 210. DB::Exception: Received from 172.16.1.6:9000. DB::NetException. DB::NetException: Connection reset by peer, while reading from socket (172.16.1.5:9000): while receiving packet from n2:9000: While executing Remote. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d17ad23 E 1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000efa5034 E 2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x000000000f13bbbc E 3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f13b071 E 4. DB::ReadBuffer::next() @ 0x000000000759da9c E 5. ./src/IO/ReadBuffer.h:0: void DB::varint_impl::readVarUInt(unsigned long&, DB::ReadBuffer&) @ 0x000000000f021016 E 6. ./build_docker/./src/Client/Connection.cpp:1061: DB::Connection::receivePacket() @ 0x0000000019421567 E 7. ./build_docker/./src/Client/PacketReceiver.cpp:66: DB::PacketReceiver::Task::run(std::function, std::function) @ 0x000000001947337e E 8. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x00000000162303df E 9. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001622fe20 E 10. ./contrib/boost/boost/context/fiber_ucontext.hpp:82: void boost::context::detail::fiber_entry_func>>(unsigned int, unsigned int) @ 0x000000001622e863 E . (NETWORK_ERROR) E (query: SELECT * FROM remote('n1', currentDatabase(), dist_secure)) helpers/client.py:239: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2024-11-05 18:13:27 [ 489 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:109, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Stdout:1 (cluster.py:117, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : No running containers (conftest.py:92, cleanup_environment) 2024-11-05 18:13:27 [ 489 ] DEBUG : Pruning Docker networks (conftest.py:94, cleanup_environment) 2024-11-05 18:13:27 [ 489 ] DEBUG : Command:['docker network prune --force'] (cluster.py:109, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Stdout:Deleted Networks: (cluster.py:117, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Stdout:roottestdotnetclient_default (cluster.py:117, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Stdout: (cluster.py:117, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:109, run_and_check) 2024-11-05 18:13:27 [ 489 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:117, run_and_check) 2024-11-05 18:13:27 [ 489 ] INFO : Running tests in /ClickHouse/tests/integration/test_distributed_inter_server_secret/test.py (cluster.py:2658, start) 2024-11-05 18:13:27 [ 489 ] DEBUG : Cluster start called. is_up=False (cluster.py:2665, start) 2024-11-05 18:13:27 [ 489 ] DEBUG : Docker networks for project roottestdistributedinterserversecret are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2024-11-05 18:13:27 [ 489 ] DEBUG : Docker containers for project roottestdistributedinterserversecret are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2024-11-05 18:13:27 [ 489 ] DEBUG : Docker volumes for project roottestdistributedinterserversecret are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2024-11-05 18:13:27 [ 489 ] DEBUG : Cleanup called (cluster.py:797, cleanup) 2024-11-05 18:13:27 [ 489 ] DEBUG : Docker networks for project roottestdistributedinterserversecret are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2024-11-05 18:13:27 [ 489 ] DEBUG : Docker containers for project roottestdistributedinterserversecret are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2024-11-05 18:13:28 [ 489 ] DEBUG : Docker volumes for project roottestdistributedinterserversecret are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2024-11-05 18:13:28 [ 489 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedinterserversecret_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:109, run_and_check) 2024-11-05 18:13:28 [ 489 ] DEBUG : Unstopped containers: {} (cluster.py:811, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : No running containers for project: roottestdistributedinterserversecret (cluster.py:825, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : Trying to prune unused networks... (cluster.py:831, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : Trying to prune unused images... (cluster.py:847, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:109, run_and_check) 2024-11-05 18:13:28 [ 489 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:117, run_and_check) 2024-11-05 18:13:28 [ 489 ] DEBUG : Images pruned (cluster.py:850, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : Trying to prune unused volumes... (cluster.py:856, cleanup) 2024-11-05 18:13:28 [ 489 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:109, run_and_check) 2024-11-05 18:13:28 [ 489 ] DEBUG : Stdout:1 (cluster.py:117, run_and_check) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup directory for instance: n1 (cluster.py:2678, start) 2024-11-05 18:13:28 [ 489 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4378, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Create directory for common tests configuration (cluster.py:4383, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Copy common configuration from helpers (cluster.py:4403, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Generate and write macros file (cluster.py:4436, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_inter_server_secret/configs/remote_servers_n1.xml', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/configs/config.d (cluster.py:4466, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/database (cluster.py:4483, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/logs (cluster.py:4494, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4579, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup directory for instance: n2 (cluster.py:2678, start) 2024-11-05 18:13:28 [ 489 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4378, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Create directory for common tests configuration (cluster.py:4383, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Copy common configuration from helpers (cluster.py:4403, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Generate and write macros file (cluster.py:4436, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_inter_server_secret/configs/remote_servers_n2.xml', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/configs/config.d (cluster.py:4466, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/database (cluster.py:4483, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/logs (cluster.py:4494, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4579, create_dir) 2024-11-05 18:13:28 [ 489 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:1d1c87eee2db', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env (cluster.py:82, _create_env_file) 2024-11-05 18:13:28 [ 489 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-11-05 18:13:28 [ 489 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-11-05 18:13:28 [ 489 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-11-05 18:13:28 [ 489 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-11-05 18:13:28 [ 489 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-11-05 18:13:28 [ 489 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', '--project-name', 'roottestdistributedinterserversecret', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/docker-compose.yml', 'pull'] (cluster.py:109, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo3 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo2 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n2 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo2 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo2 ... digest: sha256:528fcf9f160536c036... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo2 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo2 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo3 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo3 ... digest: sha256:528fcf9f160536c036... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo3 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo3 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n1 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo1 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n1 ... digest: sha256:528fcf9f160536c036... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n1 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo1 ... digest: sha256:528fcf9f160536c036... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo1 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling zoo1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n2 ... pulling from altinityinfra/integr... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n2 ... digest: sha256:528fcf9f160536c036... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n2 ... status: image is up to date for a... (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Stderr:Pulling n2 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:39 [ 489 ] DEBUG : Setup ZooKeeper (cluster.py:2720, start) 2024-11-05 18:13:39 [ 489 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/log', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/config', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/log', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/config', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/coordination', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/log', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/config', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/coordination'] (cluster.py:2721, start) 2024-11-05 18:13:39 [ 489 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', '--project-name', 'roottestdistributedinterserversecret', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--verbose', 'up', '-d'] (cluster.py:109, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:docker-py version: (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-122-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.18', 'Details': {'GitCommit': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e'}}, {'Name': 'runc', 'Version': '1.7.18', 'Details': {'GitCommit': 'v1.1.13-0-g58aa920'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-122-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestdistributedinterserversecret_default') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'ID': 'ae71819c4f5e67bb4d5ae76a6b735f29cc25774e'}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Containers': 0, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestdistributedinterserversecret_default') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.network.ensure: Creating network "roottestdistributedinterserversecret_default" with the default driver (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='roottestdistributedinterserversecret_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'roottestdistributedinterserversecret', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': '4b1d32a44c89e7d0849dd8fbbdb5d76cbc3df76fab926b2560a9311d5eb15269', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , } (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestdistributedinterserversecret', service='zoo1', number=1)} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestdistributedinterserversecret', service='zoo1', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestdistributedinterserversecret', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestdistributedinterserversecret', service='zoo2', number=1)} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestdistributedinterserversecret', service='zoo2', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo3_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestdistributedinterserversecret', service='zoo3', number=1)} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestdistributedinterserversecret', service='zoo3', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: abedbb95c5c122f50668ebceba5dbd101667ca8155b2d866285a15bcaad16363 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestdistributedinterserversecret_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Links': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Target': '/var/lib/clickhouse-keeper', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config1.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1d1c87eee2db', user='0', volumes={}, name='roottestdistributedinterserversecret_zoo1_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestdistributedinterserversecret', 'com.docker.compose.service': 'zoo1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'abedbb95c5c122f50668ebceba5dbd101667ca8155b2d866285a15bcaad16363'}, host_config={'NetworkMode': 'roottestdistributedinterserversecret_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestdistributedinterserversecret_default': {'Aliases': ['zoo1'], 'IPAMConfig': {}}}}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1d1c87eee2db') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 76f4e726a02ac968268953e2b99d540116236e3fe3a880598e6055bd7c388321 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestdistributedinterserversecret_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Links': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Source': '/clickhouse', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Target': '/usr/bin/clickhouse', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config3.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1d1c87eee2db', user='0', volumes={}, name='roottestdistributedinterserversecret_zoo3_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestdistributedinterserversecret', 'com.docker.compose.service': 'zoo3', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '76f4e726a02ac968268953e2b99d540116236e3fe3a880598e6055bd7c388321'}, host_config={'NetworkMode': 'roottestdistributedinterserversecret_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestdistributedinterserversecret_default': {'Aliases': ['zoo3'], 'IPAMConfig': {}}}}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Author': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 0fa5d58e17ea76debc11af5b47da12cfe90d7ba3248889a1f60abecd1d8155d0 (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestdistributedinterserversecret_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Links': [], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/log', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Target': '/var/log/clickhouse-keeper', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config2.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1d1c87eee2db', user='0', volumes={}, name='roottestdistributedinterserversecret_zoo2_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestdistributedinterserversecret', 'com.docker.compose.service': 'zoo2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '0fa5d58e17ea76debc11af5b47da12cfe90d7ba3248889a1f60abecd1d8155d0'}, host_config={'NetworkMode': 'roottestdistributedinterserversecret_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestdistributedinterserversecret_default': {'Aliases': ['zoo2'], 'IPAMConfig': {}}}}) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'e0c3e8953efe4bca0af432a7f3af9e09eec51891e8e6ac7eae1119f47de238bd', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('e0c3e8953efe4bca0af432a7f3af9e09eec51891e8e6ac7eae1119f47de238bd') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config1.xml', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('e0c3e8953efe4bca0af432a7f3af9e09eec51891e8e6ac7eae1119f47de238bd', 'roottestdistributedinterserversecret_default') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('e0c3e8953efe4bca0af432a7f3af9e09eec51891e8e6ac7eae1119f47de238bd', 'roottestdistributedinterserversecret_default', aliases=['e0c3e8953efe', 'zoo1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('e0c3e8953efe4bca0af432a7f3af9e09eec51891e8e6ac7eae1119f47de238bd') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '049f7a6263fc7d11048933df345e3b8c1c410d7432d9507731bf56689e05e2b1', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('049f7a6263fc7d11048933df345e3b8c1c410d7432d9507731bf56689e05e2b1') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config2.xml', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('049f7a6263fc7d11048933df345e3b8c1c410d7432d9507731bf56689e05e2b1', 'roottestdistributedinterserversecret_default') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('049f7a6263fc7d11048933df345e3b8c1c410d7432d9507731bf56689e05e2b1', 'roottestdistributedinterserversecret_default', aliases=['049f7a6263fc', 'zoo2'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('049f7a6263fc7d11048933df345e3b8c1c410d7432d9507731bf56689e05e2b1') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '637a81151dcdd0a7d10a242b964e52af0d495f69ba3fb105e1da9f8813632e2a', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('637a81151dcdd0a7d10a242b964e52af0d495f69ba3fb105e1da9f8813632e2a') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config3.xml', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:... (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('637a81151dcdd0a7d10a242b964e52af0d495f69ba3fb105e1da9f8813632e2a', 'roottestdistributedinterserversecret_default') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('637a81151dcdd0a7d10a242b964e52af0d495f69ba3fb105e1da9f8813632e2a', 'roottestdistributedinterserversecret_default', aliases=['zoo3', '637a81151dcd'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('637a81151dcdd0a7d10a242b964e52af0d495f69ba3fb105e1da9f8813632e2a') (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestdistributedinterserversecret', service='zoo1', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestdistributedinterserversecret', service='zoo2', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestdistributedinterserversecret', service='zoo3', number=1) (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_zoo3_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:119, run_and_check) 2024-11-05 18:13:40 [ 489 ] DEBUG : Wait ZooKeeper to start (cluster.py:2381, wait_zookeeper_to_start) 2024-11-05 18:13:40 [ 489 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2004, get_instance_ip) 2024-11-05 18:13:40 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_zoo1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:40 [ 489 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False (cluster.py:3141, get_kazoo_client) 2024-11-05 18:13:40 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:40 [ 489 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2024-11-05 18:13:40 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:40 [ 489 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2024-11-05 18:13:40 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:40 [ 489 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2024-11-05 18:13:41 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:41 [ 489 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2024-11-05 18:13:42 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:42 [ 489 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2024-11-05 18:13:43 [ 489 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2004, get_instance_ip) 2024-11-05 18:13:43 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_zoo2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:43 [ 489 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.1.3, port:2181, use_ssl:False (cluster.py:3141, get_kazoo_client) 2024-11-05 18:13:43 [ 489 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2024-11-05 18:13:43 [ 489 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2004, get_instance_ip) 2024-11-05 18:13:43 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_zoo3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:43 [ 489 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False (cluster.py:3141, get_kazoo_client) 2024-11-05 18:13:43 [ 489 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2024-11-05 18:13:43 [ 489 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2024-11-05 18:13:43 [ 489 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2024-11-05 18:13:43 [ 489 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2024-11-05 18:13:43 [ 489 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2024-11-05 18:13:43 [ 489 ] DEBUG : All instances of ZooKeeper started (cluster.py:2389, wait_zookeeper_to_start) 2024-11-05 18:13:43 [ 489 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env --project-name roottestdistributedinterserversecret --file /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/docker-compose.yml up -d --no-recreate') (cluster.py:3001, start) 2024-11-05 18:13:43 [ 489 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', '--project-name', 'roottestdistributedinterserversecret', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:109, run_and_check) 2024-11-05 18:13:44 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_n1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:44 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_n2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:44 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_n2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:44 [ 489 ] DEBUG : Stderr:Creating roottestdistributedinterserversecret_n1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:44 [ 489 ] DEBUG : ClickHouse instance created (cluster.py:3009, start) 2024-11-05 18:13:44 [ 489 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2004, get_instance_ip) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_n1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.1.6... (cluster.py:3016, start) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_n1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:44 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:45 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/c6ff8bdbc9d7dbdfd9f881ea9afe5e117f7781d66fca0af606c702773aea56ac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : ClickHouse n1 started (cluster.py:3020, start) 2024-11-05 18:13:46 [ 489 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2004, get_instance_ip) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_n2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.1.5... (cluster.py:3016, start) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedinterserversecret_n2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : http://localhost:None "GET /v1.42/containers/49433809172c0218df4081ec770cc58e6c50dd35d1173bf894648868dd604a05/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-11-05 18:13:46 [ 489 ] DEBUG : ClickHouse n2 started (cluster.py:3020, start) 2024-11-05 18:13:46 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n1 (cluster.py:3450, query) 2024-11-05 18:13:46 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS data_from_buffer on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query CREATE TABLE data_from_buffer (key Int) Engine=Memory() on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query CREATE TABLE dist_insecure AS data Engine=Distributed(insecure, currentDatabase(), data, key) on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure AS data Engine=Distributed(secure, currentDatabase(), data, key) on n1 (cluster.py:3450, query) 2024-11-05 18:13:47 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_backward AS data Engine=Distributed(secure_backward, currentDatabase(), data, key) on n1 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_from_buffer AS data_from_buffer Engine=Distributed(secure, currentDatabase(), data_from_buffer, key) on n1 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_disagree AS data Engine=Distributed(secure_disagree, currentDatabase(), data, key) on n1 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_buffer AS dist_secure_from_buffer Engine=Buffer(currentDatabase(), dist_secure_from_buffer, /* settings for manual flush only */ 1, /* num_layers */ 0, /* min_time, placeholder */ 0, /* max_time, placeholder */ 0, /* min_rows */ 0, /* max_rows */ 0, /* min_bytes */ 0 /* max_bytes */ ) on n1 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n2 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS data_from_buffer on n2 (cluster.py:3450, query) 2024-11-05 18:13:48 [ 489 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE data_from_buffer (key Int) Engine=Memory() on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE dist_insecure AS data Engine=Distributed(insecure, currentDatabase(), data, key) on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure AS data Engine=Distributed(secure, currentDatabase(), data, key) on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_backward AS data Engine=Distributed(secure_backward, currentDatabase(), data, key) on n2 (cluster.py:3450, query) 2024-11-05 18:13:49 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_from_buffer AS data_from_buffer Engine=Distributed(secure, currentDatabase(), data_from_buffer, key) on n2 (cluster.py:3450, query) 2024-11-05 18:13:50 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_disagree AS data Engine=Distributed(secure_disagree, currentDatabase(), data, key) on n2 (cluster.py:3450, query) 2024-11-05 18:13:50 [ 489 ] DEBUG : Executing query CREATE TABLE dist_secure_buffer AS dist_secure_from_buffer Engine=Buffer(currentDatabase(), dist_secure_from_buffer, /* settings for manual flush only */ 1, /* num_layers */ 0, /* min_time, placeholder */ 0, /* max_time, placeholder */ 0, /* min_rows */ 0, /* max_rows */ 0, /* min_bytes */ 0 /* max_bytes */ ) on n2 (cluster.py:3450, query) ------------------------------ Captured log call ------------------------------- 2024-11-05 18:13:50 [ 489 ] DEBUG : Executing query SELECT * FROM remote('n1', currentDatabase(), dist_secure) on n1 (cluster.py:3450, query) ---------------------------- Captured log teardown ----------------------------- 2024-11-05 18:13:51 [ 489 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', '--project-name', 'roottestdistributedinterserversecret', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:109, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_n2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_n1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo3_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_n2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_n1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo3_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Stderr:Stopping roottestdistributedinterserversecret_zoo1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/logs/stderr.log* || true'] (cluster.py:109, run_and_check) 2024-11-05 18:13:54 [ 489 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/.env', '--project-name', 'roottestdistributedinterserversecret', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_distributed_inter_server_secret/_instances_1/n2/docker-compose.yml', 'down', '--volumes'] (cluster.py:109, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_n2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_n1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo2_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo3_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo1_1 ... (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_n1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo3_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_n2_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing roottestdistributedinterserversecret_zoo1_1 ... done (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stderr:Removing network roottestdistributedinterserversecret_default (cluster.py:119, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Cleanup called (cluster.py:797, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Docker networks for project roottestdistributedinterserversecret are NETWORK ID NAME DRIVER SCOPE (cluster.py:776, print_all_docker_pieces) 2024-11-05 18:13:55 [ 489 ] DEBUG : Docker containers for project roottestdistributedinterserversecret are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:784, print_all_docker_pieces) 2024-11-05 18:13:55 [ 489 ] DEBUG : Docker volumes for project roottestdistributedinterserversecret are DRIVER VOLUME NAME (cluster.py:792, print_all_docker_pieces) 2024-11-05 18:13:55 [ 489 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedinterserversecret_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:109, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Unstopped containers: {} (cluster.py:811, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : No running containers for project: roottestdistributedinterserversecret (cluster.py:825, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Trying to prune unused networks... (cluster.py:831, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Trying to prune unused images... (cluster.py:847, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:109, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:117, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Images pruned (cluster.py:850, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Trying to prune unused volumes... (cluster.py:856, cleanup) 2024-11-05 18:13:55 [ 489 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:109, run_and_check) 2024-11-05 18:13:55 [ 489 ] DEBUG : Stdout:1 (cluster.py:117, run_and_check) ============================== slowest durations =============================== 22.83s setup test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users 14.12s setup test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 4.46s teardown test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users 2.39s teardown test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 0.37s call test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users 0.22s call test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total =========================== short test summary info ============================ FAILED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users SKIPPED [1] test_input_format_parallel_parsing_memory_tracking/test.py:33: Memory tracking does not make sense to check under Thread Sanitizer ======================== 1 failed, 1 skipped in 44.55s ========================= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_7dcml6 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1d1c87eee2db -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=1 --color=no --durations=0 test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.